Introduction

The importance of music on film and emotion

The full emotional effect of a movie is mainly based on the music played, in combination with the visual information. Typical movie genres are romance, comedy, horror and action.

Music shapes how we perceive the visual imagery on screen

Emotions are key elements in movies. When we think of a particular movie we’ve seen, it doesn’t take much to remember certain types of songs played in the movie. Different types of melodies, keys, instruments and many more aspects can produce a very different response in our brain.

In romantic dramas, emotions like ‘loving’ en ‘sense of longing’ are the main characteristics, but sometimes ‘sadness’ also play a role. In horror movies, fear and anxiety are the main emotions expressed in music. Dark overtones will be present. In action movies, emotions with high intensity like excitement are essential. In feelgood-comedy movies, ‘happiness’ and ‘joy’ are the main emotions, so these tracks will contain a lot of musical elements from ‘happy’ music like major tones.

The corpus for this portfolio covers a presentable selection of typical movies within four movie genres. This selection is based on the movie genre categorization of the Internet Movie Database (IMDB). Movies within an IMDB genre with typical features from other genres were excluded, as well as ‘dialogue’ tracks from the Spotify albums. The Romance/Drama playlist contains 218 tracks (12 movies), the Feelgood/Comedy playlist contains 212 tracks (14 movies), the Horror playlist has 234 tracks (10 movies) and the Action playlist has a total of 222 tracks (11 movies). Only Spotify albums from the ‘Original Motion Picture Soundtrack’ were selected.

In this portfolio there are multiple levels of analysis. First of all, individual tracks from the corpus are being analyzed with for example chromagrams, keygrams, cepstrograms, etc. After that, some analysis of the .. and lastly, a prediction analysis is done over the whole corpus.

What are the expectations?

A horror movie will be defined as a movie that seeks to scare or unsettle the audience. It is expected that this music is mostly written in the minor key. Minor chords are typically associated with sadness and melancholy. Music in horror movies wobbles and sound deliberately out of tune. For example, a lot of glissandi on violins (the screening upward). Pitch will be destabilized and pitch drops are used to stress the ‘unexpected’. One of the most iconic sounds, is the sudden sforzando tutti crash, designed to shock the audience instantly. It happens often in the midst of a musical silence, or after a pedal note.

Key features:
  • Atonality: within a track a lot of different pitches
  • Lots of different timbral components
  • Pitch destabilization (‘deformed’ sounds)
  • Slow tempo (around 80 BPM)
  • Low valence
  • Low in energy

Romantic dramas, generally, contain both ‘loving feelings’ and ‘sadness’, that is why this genre probably will oscillate between music written in both the major and minor key.

Key features:
  • Slow tempo (around 90 BPM)
  • Light tones
  • Longline and lyrical melodies
  • Medium valence: equal major and minor and low as well as high valence
  • Timbre: woodwind instruments, piano and harp

The action movie offers thrills (e.g. shooting) and spectacle (e.g. explosions).

Key features:
  • Fast tempo (120-130 BPM)
  • High staccato
  • High pitch repetition
  • Timbre: brass and percussion instruments
  • Loudness (timbre component 1)

Comedy movies are overall very happy. Happy tunes are written in the major key, are louder than other genres and probably more danceable with a high valence.

Key features:
  • High loudness
  • High in energy
  • Medium fast tempo (around 100 BPM)
  • High pitch repetition
  • Timbre: Piano, strings instruments, few harmonics
  • Loudness (timbre component 1)
  • High valence
  • Repeated instrumentation

What is the corpus of this portfolio?

Action
  • Atomic Blonde • Edge Of Tomorrow • Fast Five/Furious 7 • Hanna • John Wick • Kingsman: the Secret Service • Mad Max: Fury Road • Mission - Impossible: Fallout • Avengers: End Game • Tenet • Inception
Romance-Drama
  • Call Me By Your Name • Before Midnight • Pride & Prejudice • The Notebook • The Fault In Our Stars • Atonement • The Theory of Everything • The Age Of Adaline • Brokeback Mountain • The Guernsey Literary • After We Collided • Little Women
Horror
  • Get Out • The Lighthouse • A Quiet Place • The Conjuring • Upgrade • Split • Hereditary • Doctor Sleep • 10 Cloverfield Lane • IT
Feelgood-Comedy
  • This Is the End • They Came Together • Bridesmades • Hunt For the Wilderpeople • The 40 Year Old Virgin • The Gig Sick • Scott Pilgrim vs. the World • 21 Jump Street • Girls Srip • Crazy Rich Asians • American Hustle • Pitch Perfect • Wet Hot American Summer • Napoleon Dynamite

In this small study these questions will be answered: * In what way do typical tracks from different movies differ? + Emotional quadrant + Tempo

  • What are the features that distinguish movie genres the most?

  • Self-similarity matrices

  • Chromagrams and keygrams

  • Cepstrograms

  • Timbral coefficients

  • Is it possible to predict movie genre based on just film music?

  • Confusion matrices

  • Forest model

Spotify features

1. In what way does the film music from different movie genres differ, based on the emotional quadrant?


Select movie genre:


Mode 0 = minor, mode 1 = major

This graphic shows the emotional quadrant of tracks played in movies from movie genres Horror, Action, Feelgood/Comedy and Romance/Drama with color representing the mode and size the loudness of the track. Valence describes the musical positiveness, energy describes the arousal. Overall, louder songs are in the Happy/Joyful section. There is a clear distinction of the Horror and Action genres from the other two genres, the majority of the tracks is displayed at very low valence values. Most of the tracks from Horror movies are concentrated in the Depressing/Sad section of the emotional quadrant graph, with a lot of minor songs and these are overall very ‘quiet’. Tracks from Action movies are more smeared out, but locate mainly in the Angry/Turbulent and Depressing/Sad sections of the graph. Surprisingly, the majority of these tracks do not seem to be very loud at all, which is in contrast with the expectations. Feelgood-Comedy tracks are more scattered, but in comparison with the other genres, this genre has a lot of tracks in the Happy/Joyful section with more louder songs, which is in line with the expectations. The tracks of Romantic/Drama’s are localized mainly throughout the upper right and bottom left of the plot, but do have a low valence overall.


Select specific movies:

2. What about tempo? Is there a difference between genres and minor & major keys?


Density plots

These density plots show the distribution of tempo in Beats Per Minute (BPM) for the movie genres. On average, Feelgood/Comedy movies have the highest BPM. The average BPM for Horror movies is the lowest (89 BPM). The distribution for Romance/Drama and Action is about the same. It is also clear that overall, songs in minor key do have a slightly lower BPM than major-key songs, especially in Action movies.

The expectation was that the Action movie has the highest overall tempo from all genres. An explanation for this result is that the average Action movie has both very uptempo tracks, but also very quiet en slow tracks. Slow ‘Horror-like’ tracks are used to build up the tension. That is why the distribution for major Action tracks is quite wide. The tempo distribution for minor tracks in Action movies is also a surprising result. Apparently the songs written in a minor key are the generally slow Horror-like tracks.


Median values for tempo
category mode median
Action Major 117.594
Action Minor 98.890
Feelgood/Comedy Major 125.056
Feelgood/Comedy Minor 116.313
Horror Major 88.599
Horror Minor 90.337
Romance/Drama Major 108.105
Romance/Drama Minor 101.931

3. Self-similarity matrices of typical comedy and romantic tracks ‘Feelgood’ and ‘Another Dance’


Self-similarity matrices

In a self-similarity matrix each element of the feature sequence is being compared with all other elements. Path-like structures represent exact repetitions. There is one main diagonal visible, this is because both axis represent the exact same song. Block-like structures represent homogeneous regions. This is where music features stay somewhat constant over the duration of an entire musical part.

The left visualizations represent self-similarity matrices for ‘chroma’. It demonstrates at which points in the track the same pitches occur. The right visualizations represent the same song, but with ‘timbre’, also referred to as ‘tone color’. Later on there will be more information about this musical feature.

####‘Feelgood’

is a typical Feelgood/Comedy track (what else!) from ‘They Came Together’. The SSM of chroma shows a block-like pattern. At t = 25 some percussion instruments and a piano come to the fore. In the SSM of timbre the last section is very distinguishable. It is clear that a different instrumentation is used. The lead singer stops and a electric guitar starts playing.

‘Another Dance’

is a track from the Romantic/Drama ‘Pride and Prejudice’. This is a good example of exact repetitions. Only diagonal lines are visible, but no big block-like structures. Lines that are diagonal even to the main diagonal are exact repetitions. It is very clear when you listen to this track.


Judge yourself!

4. The differences between a typical Horror track a typical track from a Action movie based on chroma features


Chroma- and keygrams

Both left grams represent chromagrams. These sum up all pitch coefficients that belong to the same chroma, so this gram cyclic in nature. The graphics to the right represent chordograms with all chords used in the track.

‘Curse Your Name’

is a Horror track from the movie ‘The Lighthouse’. This track is written in the minor key and has an extremely low valence (0.021). This track has both sections with long lasting pitches and sections with a lot of pitches played at the same time (pitch mix). This is why the track doesn’t sound harmonically ‘correct’ to the human ear, just like the usage of very high and low pitched sounds. However, in these grams high and low pitch coefficients are not distinguished because of the cyclic nature. These characteristics make it a very typical Horror track. The sections are accentuated with white vertical lines.

‘Change of Plan’

A typical feature from the Action genre is a build up, to build up some tension. This build up is visible in the chromagram till (t = 0 - t = 53). This change is also associated with a change in keys (see chordogram). There is a fast alternation of different pitches throughout the whole track which is also typical in a Action movie.


5. The differences of timbral components of an outlier and a typical track in horror film-music


Cepstrograms: Timbre components

Timbre, also known as “tone color”, is the perceived sound quality of a musical note or sound. It distinguishes various types of musical instruments. There are twelve timbre coefficients in total. The values are high level abstractions of the spectral surface ordered by degree of importance.

  • c01: The ‘average loudness’,
  • c02‘brightness’. Increased levels of mid and high frequency content are referred to as ‘brighter’.
  • c03: Closely related to the ‘flatness’ of a sound. A high flatness indicates that the spectrum has a similar amount of power in all spectral bands (i.e. similar to white noise) and low flatness indicates a “spiky” spectrum (mixture of sine waves).
  • c04: Sounds with a ‘stronger attack’.

Doll box is different than the majority of the horror film music. It has a extremely high valence (0.854) for Horror film music. This track is being compared with Curse Your name, which is a very typical horror track with an extremely low valence (0.021). The energy values are more or less the same (0.0625 and 0.0725). Doll box is written in a major key, with very high pitched, distinctive tones.

In Doll Box it is very clear which timbre components are being used in this track; c02, c03 and c04. The sound of this track represents the typical sound of a doll music box. Because these three components are very constant throughout the whole track, it is hard to distinguish the different sound characteristics. Two clear sections in the cepstrogram are the very contrasting parts at t = 1 and at t = 32. These parts are riffs on a copper xylophone.

As you can see, the timbral components of Curse Your Name are much more spread out in the cepstrogram. This is very typical in horror music, because horror music tends to have a lot of different musical characteristics/instruments played at the same time, that seeks to give the audience an uncomfortable and unsettling feeling. The first bright yellow part of co2 is very distinguishable in this track. It represents a wind-instrument, probably a trumpet. Immediately after this part, a somewhat longer c01 appears. This part is a very low-pitched string-instrument, it sounds like a string bass. The yellow part of c05 is a very sharp sound, a high-pitched flute which is very unpleasant to the ear. It is clear that both very high- and low pitched sounds are being used at the same time in typical horror music.

Comparison of c03 in both tracks: this sound is in both tracks a very high-pitched instrument, however, in doll house this sound is made by a percussion instrument. In Curse Your Name, this sound sounds more like a wind instrument.

7. The mean differences of timbre coefficients of each movie genre


The first 60 tracks from each genre are used

When comparing the twelve Spotify timbre coefficients between the four movie genres, the main difference is in the second and third coefficient. For c02, horror music really differs from the other genres. It seems that the range is way more in the positive area. This is probably due to the use of stringed instruments in horror music.

Classification

1. Is it possible to predict movie genre based on only music score?


The analysis is performed over the first 120 tracks from each genre category

Genre classification with all four genres was performed using support vector machines in a ten-fold cross-validation test. Here, you can see three confusion matrices; ‘all features’ (track-level-, timbral- and chroma features), ‘timbral features’ and ‘chroma features’ are being compared. The darker the color grey, the better the prediction. Overall, the chroma features performed the worst among the classification tasks, all features combined predicted movie genre the best.

All features

All features provided the highest genre accuracies for Romance/Drama From the confusion matrix, it is clear that there is a clear diagonal dark-grey line, from the upper left to the bottom right. This means, that the model predicted these the best.

Timbral features

Feelgood/Comedies are most often confused with Action movies for timbral features. This means that the timbral features of Feelgood/Comedy tracks some what similar are to those of Action movies. An explanation is that both Action and Feelgood/Comedies are overall very ‘loud’. This loudness is present in timbral coefficient c01. Timbral features from Horror and Feelgood/Comedies are the most distant from each other. Because the definition of timbral coefficients is somewhat arbitrary, we cannot really conclude what features these exactly are.

Chroma features

Chroma features (pitches) do not seem to predict movie genre very well. Music from Horror movies is the least distinguishable when looking at chroma features.

A striking observation is that from the true Horror movies, only 25% was correctly categorized as a ‘Horror track’. This is exactly at chance level.

It looks like Romantic/Drama tracks do differ on chroma features. So these songs do have some specific chroma features that most songs contain.


Precision & recall all features
class precision recall
Action 0.5203252 0.5333333
Feelgood/Comedy 0.5794393 0.5166667
Horror 0.6699029 0.5750000
Romance/Drama 0.5374150 0.6583333
Precision & recall timbral features
class precision recall
Action 0.4462810 0.450
Feelgood/Comedy 0.4766355 0.425
Horror 0.6160714 0.575
Romance/Drama 0.5357143 0.625
Precision & recall chroma features
class precision recall
Action 0.4242424 0.4666667
Feelgood/Comedy 0.3511450 0.3833333
Horror 0.3894737 0.3083333
Romance/Drama 0.5491803 0.5583333

Even though chroma features do not really differ across Horror and Feelgood/Comedy movies, these genres are the most distant when looking at the confusion matrix for all features combined. This makes sense, because Comedies are in general very happy and uptempo, while Horror movie tracks are overall very slow and depressing.



Precision recall from decision tree
class precision recall
Action 0.5773196 0.4666667
Feelgood/Comedy 0.5853659 0.6000000
Horror 0.6376812 0.7333333
Romance/Drama 0.6885246 0.7000000

From the decision tree classification, the precision and recall for the different genres are much better than the ones for the confusion matrices. Recall (sensitivity) is the amount of tracks that were correctly categorized to the right genre. The recall for the horror tracks is best, with a total accuracy of 75%. So 75% of the tracks that belong to Horror films, are correctly categorized. The precision and recall is lowest for Action movies.

2. What musical features really do characterize movie genre?


Forest model

From the forest model, it is very clear that there are some features that really characterize the four movie genres. The top 6 features are:

Track level features

  • Valence
  • Acousticness
  • Danceability
  • Energy

Timbral components

  • c06
  • c01

It seems that timbre c06 is an important component in distinguishing movie genres. However, because timbre components above c04 are very hard to identify, it is not very clear what aspect this exactly is.

In the next slide, we’ll perform a new analysis on these 8 components.

3. Confusion matrix with characterizing components


When looking only at the top 8 components from the previous page and compare it with the other:


Precision recall for selected features
class precision recall
Action 0.5368421 0.4250000
Feelgood/Comedy 0.4850746 0.5416667
Horror 0.5350877 0.5083333
Romance/Drama 0.4890511 0.5583333
Precision recall for all features
class precision recall
Action 0.5203252 0.5333333
Feelgood/Comedy 0.5794393 0.5166667
Horror 0.6699029 0.5750000
Romance/Drama 0.5374150 0.6583333

Final thoughts

Final thoughts

The results support the notion that high intensity movies like action and horror, have musical cues that are measurably different from the scores of movies with more measured expression of emotion, like comedy and romance.

This small study presents a preliminary examination on a corpus of music collected from film scores in four genres, from a total of 47 movies (Action, Romance/Drama, Horror and Feelgood/Comedy) utilizing all kinds of music representations from track-level-features, to chroma and timbre self-similarity matrices, musical keys, and tempo.

From the emotional quadrant

Initial results suggests that the music from

However, even when using very distinct movie genres, it is clear that such a labeling scheme is likely too broad as several tracks within a specific genre may exhibit characteristics of music from another genre. For example, we’ve seen that music from Feelgood/Comedies are musically very much similar to these of Action movies, when looking at timbral features.

A more close examination of each individual track will probably serve to improve classification accuracy.

The creator

This portfolio was made by me, Iris, a psychology student who follows this course ‘Computational Musicology’ as part of the Minor Artificial Intelligence. I especially wanted to choose a subject that was somehow related to the field of psychology. But how would I incorporate this in such a computational course, without participants? After some brainstorming I came up with the idea of using film music. I really enjoyed making this portfolio. I had very minimal R skills from my bachelor Psychology. I learned a lotttt about how to code in R, which was sometimes a challenge, but I’m very happy with the results. I hope you enjoyed reading this portfolio about film music!